[SQL][DML] Resolve Path Based Tables for both Reads and Writes by andreaschat-db · Pull Request #56039 · apache/spark

andreaschat-db · 2026-05-21T13:53:57Z

What changes were proposed in this pull request?

This PR fixes an issue where we would only correctly resolve path based tables if they were the target of transactional writes. The DataSource catalog resolution is being moved to CatalogAndIdentifier so it can by used for both the target of the transactional write as well as the any other tables read. The approach maintains exactly the same semantics as the Dataframe path based identifier resolution.

Why are the changes needed?

The catalog needs to be resolved correctly for all path based tables participating in a transactional write. This includes both the target as well as any other tables read. Without this fix, any scan on path-based tables would miss the configured catalog and would fall back to the session catalog.

Furthermore, in DSv2 transactions it is important to restrict all participating tables to a single catalog. This allows the single transactional catalog to track all reads and predicates that are relevant to the transaction.

Does this PR introduce any user-facing change?

Yes. Path based tables are now correctly resolved for both read and write operations.

How was this patch tested?

Added new tests.

Was this patch authored or co-authored using generative AI tooling?

Claude Sonnet 4.6.

huaxingao · 2026-05-27T21:12:54Z

@aokolnychyi @andreaschat-db

Is this PR aiming for 4.2?

andreaschat-db · 2026-05-28T07:48:15Z

@aokolnychyi @andreaschat-db

Is this PR aiming for 4.2?

Yes. It is an important fix for the DSv2 Transaction API.

huaxingao · 2026-05-28T16:16:17Z

@aokolnychyi @andreaschat-db
Could you share when you expect this PR to be merged and backported to branch-4.2? Also, should we treat this as a blocker for cutting Spark 4.2.0 RC1?

aokolnychyi · 2026-05-28T20:52:07Z

+   * resolution.
+   */
+  def supportsCatalogOptionsResolver(conf: SQLConf): DataSourceCatalogResolver =
+    (nameParts: Seq[String]) =>


Should we attempt to do this only if we have 2 name parts?
We are talking about SQL on file scenarios, right?

Do we care about spark.sql.runSQLonFiles? See ResolveSQLOnFile.

Should we attempt to do this only if we have 2 name parts?
We are talking about SQL on file scenarios, right?

Yes these are SQL on file scenarios. The 2 name part sounds like a limitation of v1? IIUC in v2 dataframes we do not have such restriction. I guess the connector can decided how to handle it in extractCatalog/extractIdentifier?

Do we care about spark.sql.runSQLonFiles? See ResolveSQLOnFile

I added the sentinel.

aokolnychyi · 2026-05-28T20:59:30Z

+            // identifier head, fall through to currentCatalog and let later analysis raise
+            // table-not-found. This matches the v1 file-format precedence (catalog first,
+            // path-based as fallback).
+            Option(catalogManager.catalogAndIdentForDataSource(nameParts)).flatten match {


I think this introduces a slight but problematic behavior change.

Say a user has a database named delta in their session catalog with a table orders.

SELECT * FROM delta.orders

Before: resolves to (session_catalog, ns=["delta"], name="orders") → finds the table correctly.

After: if the delta data source implements SupportsCatalogOptions, the SCO resolver intercepts, calls extractIdentifier with options derived from {"path": "orders"}, and routes to whatever catalog extractCatalog returns.

It seem like we should only pick the SQL on file approach if we KNOW / TEST that the current catalog can't resolve the requested namespace / identifier pair.

Yes thank you. I fixed the resolution order.

aokolnychyi · 2026-05-28T23:14:19Z

I think this one is a blocker because we introduced partial SQL on file support in DSv2 that is incorrect, not only within the transaction itself.

aokolnychyi · 2026-06-02T04:49:13Z

@huaxingao @andreaschat-db, I spent two days thinking about this PR. The current implementation still has a correctness issue but I was not sure how to fix it. I have an idea I will try tomorrow and will update here.

Sorry it is taking longer, it ended up being a tricky problem.

aokolnychyi · 2026-06-03T02:06:46Z

Let me share my thoughts:

I am worried about using CatalogAndIdentifier in the current way as it is too broad. It applies to functions, procedures, etc. We only care about tables in this case.
Our SupportsNamespaces guard is not reliable. Even its doc says the class is optional. This means table catalogs can load tables within namespaces without implementing this trait. Even knowing the namespace exists, doesn't tell us whether it is a valid identifier or not.
The SupportsCatalogOptions was designed for DataFrame writer/reader cases, we adapt it for use in SQL on path without any additional opt-in mechanisms.

Do we actually have to teach Spark to resolve SQL on path in DSv2? In theory, connectors like Delta already override the session catalog (spark_catalog), so can't we simply support path identifiers within those connector catalogs?

aokolnychyi · 2026-06-03T02:10:26Z

If we can support SQL on path in custom session catalogs like DeltaCatalog, then we will only need to rollback changes that we added for SQL on path in DSv2 and it should be enough to unblock 4.2.

aokolnychyi · 2026-06-03T02:10:58Z

@andreaschat-db, what do you think? Can we support SQL on path on the Delta side? If we do that, we can start transactions pretty easily as I assume DeltaCatalog would implement the new transactional API.

We already have SupportsCatalogOptions for non-SQL transactions. We only care about SQL cases here.

andreaschat-db · 2026-06-03T13:57:46Z

@andreaschat-db, what do you think? Can we support SQL on path on the Delta side? If we do that, we can start transactions pretty easily as I assume DeltaCatalog would implement the new transactional API.

We already have SupportsCatalogOptions for non-SQL transactions. We only care about SQL cases here.

My understanding is that yes we can do this on the connector (Delta) side. The requirement is that for path based tables the current catalog needs to be a transactional catalog that can also understand the path. IIUC this is consistent today with how delta works. If the current catalog is set to a non-delta catalog, delta paths won't work.

The idea in this PRs was to have a more consistent handling of paths in Spark since we are already doing this in Dataframes. There are advantages of doing that (uniform handling across connectors, consistency with Dataframe handling, catalog extraction that independent of the current catalog etc) but I am ok of pushing this to the connector.

Regarding the concerns above:

We could narrow down the scope by creating CatalogAndIdentifierWithPathBasedResolution. This does whatever CatalogAndIdentifier plus the path based resolution. Then can cherry pick which call sites use the former and which the latter.
To guard against SupportsNamespaces being optional, we can change the resolution order so that if the current catalog does not implement SupportsNamespaces, never tries to resolve via the data source code path. Therefore, the behaviour remains the same for connectors that do not adapt their implementation. Connectors that want to support path based resolution need to also implement SupportsNamespaces.
We could introduce a new trait on top of SupportsCatalogOptions. Only connectors that implement the new trait get the path resolution in SQL. This also separates the intent between supporting SCO in Dataframes and SQL.

aokolnychyi · 2026-06-03T15:07:30Z

Yeah, I would be open to explore a better solution in Spark. What if we do this after 4.2? Can we for now remove the portions of the path-based resolution we added for transactions to unblock 4.2?

andreaschat-db · 2026-06-04T07:06:59Z

This PR was succeeded by #56316.

### What changes were proposed in this pull request? Current implementation for path based tables in SQL is partial. We are removing it for Spark 4.2. ### Why are the changes needed? The previous [attempt](#56039) to add path-based table support was halted due to concerns. For Spark 4.2 we are letting the connector handle it. ### Does this PR introduce _any_ user-facing change? No. Path based support in SQL was not yet released. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? Claude Opus 4.7. Closes #56316 from andreaschat-db/dsv2TransactionRemoveSQLPathBasedSupport. Authored-by: Andreas Chatzistergiou <andreas.chatzistergiou@databricks.com> Signed-off-by: Anton Okolnychyi <aokolnychyi@apache.org>

### What changes were proposed in this pull request? Current implementation for path based tables in SQL is partial. We are removing it for Spark 4.2. ### Why are the changes needed? The previous [attempt](#56039) to add path-based table support was halted due to concerns. For Spark 4.2 we are letting the connector handle it. ### Does this PR introduce _any_ user-facing change? No. Path based support in SQL was not yet released. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? Claude Opus 4.7. Closes #56316 from andreaschat-db/dsv2TransactionRemoveSQLPathBasedSupport. Authored-by: Andreas Chatzistergiou <andreas.chatzistergiou@databricks.com> Signed-off-by: Anton Okolnychyi <aokolnychyi@apache.org> (cherry picked from commit 42db152) Signed-off-by: Anton Okolnychyi <aokolnychyi@apache.org>

aokolnychyi · 2026-06-04T18:06:48Z

The alternative PR has been merged into master, branch-4.x, and branch-4.2. Thanks!

andreaschat-db force-pushed the dsv2TransactionPathBasedFix branch from 7a0af9a to bf43ee1 Compare May 22, 2026 14:03

andreaschat-db marked this pull request as ready for review May 22, 2026 14:03

andreaschat-db added 18 commits May 28, 2026 08:20

Experiment v1

f678ac8

fixes

b2cabf0

Improvements

b37b6c3

DataSourceCatalogResolver

19a70d6

Remove path based from QE

72c9169

Improvements

cc2f1cc

v1 view fix + cleaning

bdb8c10

Move nullability guard back to LookupCatalog

49ad3bb

Improvements

591ddf4

Delegate identifier parsing to the connector for path based tables

e72602e

Move Fake path based providers to FakeV@Provider

6962d07

Remove single catalog enforcement

a6c51ca

Improvements

51a96a4

Fix FakeV2Provider

673b82d

Add PathBasedTableSuite

f751c61

Improvements

092f1d9

Change resolution precedence + test improvements

05718e9

Fix multipart resolution + test nits

1454dca

andreaschat-db force-pushed the dsv2TransactionPathBasedFix branch from bf43ee1 to 1454dca Compare May 28, 2026 08:47

aokolnychyi reviewed May 28, 2026

View reviewed changes

Address comments

d7c440b

andreaschat-db requested a review from aokolnychyi May 29, 2026 15:12

andreaschat-db marked this pull request as draft June 4, 2026 06:40

andreaschat-db changed the title ~~[SPARK-56695][SQL][DML] Resolve Path Based Tables for both Reads and Writes~~ [SQL][DML] Resolve Path Based Tables for both Reads and Writes Jun 4, 2026

andreaschat-db mentioned this pull request Jun 4, 2026

[SPARK-56695][SQL][DML] Remove Path Based Table support in SQL #56316

Closed

Uh oh!

Conversation

andreaschat-db commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

huaxingao commented May 27, 2026

Uh oh!

andreaschat-db commented May 28, 2026

Uh oh!

huaxingao commented May 28, 2026

Uh oh!

aokolnychyi May 28, 2026

Choose a reason for hiding this comment

Uh oh!

aokolnychyi May 28, 2026

Choose a reason for hiding this comment

Uh oh!

andreaschat-db May 29, 2026

Choose a reason for hiding this comment

Uh oh!

aokolnychyi May 28, 2026

Choose a reason for hiding this comment

Uh oh!

andreaschat-db May 29, 2026

Choose a reason for hiding this comment

Uh oh!

aokolnychyi commented May 28, 2026

Uh oh!

aokolnychyi commented Jun 2, 2026

Uh oh!

aokolnychyi commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aokolnychyi commented Jun 3, 2026

Uh oh!

aokolnychyi commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreaschat-db commented Jun 3, 2026

Uh oh!

aokolnychyi commented Jun 3, 2026

Uh oh!

andreaschat-db commented Jun 4, 2026

Uh oh!

aokolnychyi commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andreaschat-db commented May 21, 2026 •

edited

Loading

aokolnychyi commented Jun 3, 2026 •

edited

Loading

aokolnychyi commented Jun 3, 2026 •

edited

Loading